Lexical Semantic Association between Web Pages – a Lexical Knowledge Based Method

The candidate confirms that the work submitted is her own and that appropriate credit has been given where reference has been made to the work of others i ACKNOWLEDGEMENT I would like to express my gratitude to my supervisor Eric Atwell, Bill Whyte for helpful commentary and suggestions and also to Clive Souter, George Demetriou who provided me with LKB dictionary and gave me good suggestions. I also would like thank my teachers Paul & who always encourage me. Without their constant support and encouragement, I can't fulfil this thesis. I want to express my respect and thankfulness to my dear parents. ii ABSTRACT The rich variety of knowledge available on the World Wide Web makes it an attractive target for data mining and also language processing. In this project, a linguistic method, which was developed to measure the lexical semantic association between two words, is adapted to the task of measuring the semantic similarity between web pages. This lexical knowledge based method could also be used for meaning trend representation or theme representation. Themes are connotative meanings separated or contained in the textual units, such as text, or web pages, and are difficult to represent quantitatively and properly. This work also tries to propose a scientific description of themes that are viewed from the point of lexical semantics. Once textual units have been tagged with Lexical Semantic Tags contained in the Lexical Knowledge Base, themes within the units can be generated. The semantic representation of each web page is a bag of words after tagging which is believed contains certain themes. A program implements the measurement of lexical semantic similarity between two web pages. Various experiments have been undertaken to test the impact of text distance, noisy words and text length. To assess the precision of the methodology, we compare the result of the system with existing commercial information retrieval system and human judgements. A theme space was created to support the evaluation.